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DESCRIPTIQN 



Elastic image legistratioii 

The present invention relates to the jBeld of digital imftgitig ^ for exanq>le, in the field of 
5 medical imaging. In particular, the present invention relates to amethod of registering a 
first image (e.g. a floating image) and a second image (e.g. a reference image), an 
image processing device and to a computer program for registering a first image and a 
second inoage. 

10 When two images of the same object have been taken firom different projections or at 
different points of time, or during different moving stages of the object of interest^ or 
when two images do not come jfrom the same object, but firom objects of the same class 
(for example, two &ces or two brains), it may be hi^y desirable to establish 
correspondences. Two classes of deformations have to be compensated in order to 

15 establish &ese correspondences: ri^d deformations resulting from translational and 
rotational movements of the object imaged and non-rigid deformations resulting from 
elastic deformations of the object imaged. An example of such an plication is to 
match a brain with an anatomical atlas in order to automatically find abnormalities or to 
label the brain with anatonodcal names. Another example is to observe the deformation 

20 over the time of, for exaxxxple, a beating heart or to register images of a deformable 

region of a patient (for example, abdomen) taken at different times, Anotiier example is 
the combination of images acquired with difi^ient modalities such as Computer 
Tomogr^hy (CT), Magnetic Resonance Imaging (MRT), Ultra Sound (US), Positron 
Emission Tomogr^hy (PET), Single Photon ^nission Computed Tomogr^hy 

25 (SPECT) to improve the diagnostic iirocess through the combined analysis of 
anatomical and physiological information. 



It is an object of the present invention to provide for an improved image registratioti. 
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According to an exemplary embodiment of tihie present invention as set forth in claim 1, 
the above object may be solved by a method of registering a first itnage and a second 
image, wherein at least one first landmadc is selected in the fibtst inoiage and at least one 
second landmark is selected in the second image. Then^ these first and second images 
5 aieregisteredbyusingasimilarityvalue, wMch relates to a similarity of a fira 
Ihe first image determined by at least one first landmark and a second region in the 
second image determined by at least second landmark. The first and second landmarks 
correspond to each other. 

10 In other words, according to this exemplary embodiment of the present invention, 
landmarks, for example in medical applications anatomical landmarks, are 
automatically or manually selected in the images. These landmarks are used to define 
corresponding regions in the images. For example, in angippr^hic images relating to 
different heart phases, cross points of the LAD (left anterior descending) and tiie LCX 

15 (left circmiiflex) may be used as landniarks in the differ Then, a 

similarity value is determined to express a similarity between Ihe region indicated by the 
first landmajdcCs) in Ihe first image and the region indicatedby the second landmark(s) 
in the second image. This similarity value is used to register the images. 

20 — eorresponding4andmarfcs in^he-first-and^eeondimages-are landmM 

example, for anatomical ^plications, mark the same or corresponding anatomical 
structures in the first or second images. Le. corresponding landmarks in the first or 
second images are points or regions ideally to be registered on each other. 

25 According to another exerr^lary embodiment of the present invention as set forth in 
claim 2, the first and second landmarks are selected in accordance with aquali^dng 
function. This means that, for example, only the most significant landmarks, i.e. only 
the most significant corresponding features are selected as landmarks. In other words, 
this exeinplary embodiment of the present invention focuses the computational work on 

30 few but significant landmarks, for example, in areas where needed due to larg^ and 
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hig^y non-linear deformations and/or well delineated anatomical structmes allowing 
for unambiguous and precise registration. It should be noted that the landmarks may be 
selected interactively or automatically. 

Advantageously, according to this exemplary ^bodiment of the present invention, the 
speed and robustness of the image registration may be improved by concentrating only 
on significant landmarks and image areas showing highly non-linear deformations. 

According to another exemplary raobodiment of the present invention as set forth in 
claim 3, a local deformation is determined for each landmark in the first and second 
images. These local deformations are expanded into a global deformation for the 
images. The global deformation may be used to perform the registration. 

According to another exemplary embodiment of the present invention as set forth in 
claim 4, a similarity value is determined to describe a similarity between respective 
regions in the first and second images defined by the landmarks in the first and second 
images. This is done by using the global deformation fields. Advantageously, by this, a 
similarity of defined regions in the first and second regions maybe determined and 
regions may be identified where the similarity is not yet sufficient 

According to another exemplary embodiment of the {^resent invention as set forth in 
claim 5, for regions where the similarity is not yet suflBcient, a new landmark is defined 
in the first and second image and the above described method may be iteratively 
repeated, i.e. for the new sub-regions defined by the new landmark, new sub-similarities 
with sub-similarity values are determined by using a local deformation field of this new 
landmark, which is expanded to a global deformation field for the image. Then, all the 
newly defined sub-regions are tested as to whether a respective similarriy value meets 
with a pre-set threshold and if not, according to this exenxplary embodiment of the 
present invention, new landmarks or new regions may be iteratively defined. 
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Due to the fact that the lejBBement process is always limited to the region with 
insufficient similarity and is not applied to the whole images, the computational efforts 
maybe reduced. Furthermore, by either automatically or interactively performing this 
iterative refinement it allows for a very accurate registration. 

5 

According to another exeniplary embodiment of the present invention as set forth in 
claim 6, the method is iteralively repeated until all similarities of all regions and all sub* 
regions exceed a pre-set threshold value. 

10 According to another exemplary embodiment of the present invention as set forth in 
claim 7, the method is applied in medical imaging on one of CT data sets, MRI data 
sets, PET data sets, SPECT data sets, andUS data sets. 

According to another exemplary embodiment of the present invention as set forth in 
15 claim 8, an image processing device is provided, allowing to perform a non-rigid image 
registration method by an iterative refinement process based on landmarks and a 
similarity value such as a gray value based measurement of a region in the images. 

Advantageously, the image processing device according to this exemplary embodim^t 

"20^ — of-the-present4nventien4ias-an-kaproved-reg[stm1aon-s^ 

accurate and mbust registration of the first and second images. 

According to another exemplary embodiment of the present invention as set forth in 
claim9, a cotxq)uter program is provided for registering a first image and a second 
25 image. The computer program maybe written ia any suitable iHX>gramming language, 
such as C++ and may be stored on a compater readable device, such as a CD-ROM. 
However, the conq)uter program according to the present invention may also be 
presented over a networic such as the WorldWideWeb, firom which it may be 
downloaded, for example, into the internal working memory of a processor. 



30 



-5- 



PHDE030412EPP 



It may be seen as the gist of an exsmphay embodiment of ibe present invention Ibat a 
non-rigid image registration is performed by an iterative refinement process based on 
landmarks combining local template matching. An example for a local tenq)late 
matching is described, for example, in P. Rosch et al, "3D respiratory motion 
compensation by template propagation" in T. DoM andR. Kildnis, editors, "Proceedings 
of Ihe 5* international conference on medical image computing and computer-assisted, 
intervention" - MICCAI 2002, pages 639-646, Springer, 2002, which is hereby 
incorporated by reference. 

The image similarity in fbe respective regions may, for example, be determined in 
accordance with J. V. Hajnal et al, editors, "Medical image registration", CRC Press, 
2001, which is hereby incorporated by reference. The method focuses ibe computational 
work on few but significant landmarks and automatically refines the deformation field 
only in areas where needed due to large and highly non-linear deformations and/or weU 
delineated anatomical structures allowing for unambiguous and precise registration. 
When A denotes an image that has to be registered with a reference image B, starting 
fomafcwbutsignificantlandmarics,hnage^istiledwiflisiinpUcesP^ spannedby 
rffl dimensionality of llie data set) neighboring landmarics. A global deformation 
field D is constructed by locating Ibe landmarics in B based on local (for instance afBne) 
deformations m their neighborhoods. Every simplex (region defined by the landmarics) 
Pa is transformed into a simplex Pb = D(Pj) and the similarity of, for exan^le, flie gray 
values of image A lying in Pa and the gray values of image B lying in Ps is estimated 
using, for example, mutual information or cross-correlation. If and Pb are not 
sufficiently similar, a new landmark is located in Pa and Pa is divided into a pluraHty of 
<*f 1 smaller sinqilices. Advantageously, due to tihis, the tiling is iteratively refined untfl 
a desired similarity is adrieved, resulting in a tiling grid that automatically ad^ts itself 
to local non-linearities and, for example, to anatomical structures. 

These and other aspects of the present invention will become parent fixim and 
elucidated with reference to the embodiments described hereinafter. 
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Exemplary embodiments of the present invention wiU be described in the following, 
with reference to the following drawings: 

Fig. 1 shows a schematic representation of an image processing device according to an 
5 exemplary embodiment of the present invention, ad^ted to execute a method accordmg 
to an exemplary embodiment of the present invention. 

Fig. 2 shows a sin5)lified flow-chart of an exemplary embodiment of a method 

according to the present invention, which may be performed in the image processing 
10 device depicted in Fig. 1. 

Figs. 3 to 6 show simplified sketches of images and indicated operations performed on 
the images according to an exenq)laty embodiment of the method according to the 
present invontion. 

Fig. 7 shows two images as exan^les of aglobal elastic registration during a respiratory 
motion of the respiratory system performed in accordance with the exemplary 
embodiment of the method according to the present invention. 

20 Fig. 1 depicts an exranplaiy embodiment of an image processing device according to the 
present inven tion, for ex e cuting an exemplary embodiment of amet hod in accordanrft 



15 



with the present invention. The image processing device depicted in Fig. 1 comprises a 
central processing unit (CPU) or image processor 1, connected to a memory 2 for 
storing images or similarity values used or generated during the method. The image 
25 processor 1 maybe connected to aphirality of input/outpuli network or diagnosis 

devices, such as an MR device or a CT device. Furthermore, the image processor 1 may 
be connected to an ultrasonic scanner. The image processor is iurthermore connected to 
a display device 4 (for example, a conaputet monitor) for displaying information or 
images computed or adapted in tiie image processor 1 . An operator m^ interact with 

30 theimageprocessorlviaakeyboard5and/orotiieriiiput/ou^utdevices,whicharenot 
depicted in Fig. 1. 
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Fig. 2 shows a flow-chart of an exeiiq>laiy embodiment of a method fiw registering a 
first image and a second image. 

La the following, the present invention will be described with reference to medical 
5 imaging, where the landmarks relate to anatomical features in the images and/or to areas 
having large and highly non-linear deformations and/or well-delmeated anatomical 
stractures allowing for an unambiguous and predse registration. However, it should be 
noted that the present inveoition may also be ^lied to non-medical plications, for 
example, in material testing or quaKty control, where images of actual products are 
10 con5>ared to reference images. Also, this technique inay be used for image recognition 
systems, for example, where faces filmed are to be matohed wifli pre-registered feces to 
identify the feces or persons. 

In short, the method according to this exemplary embodiment of the present invention, 

15 described in the following with reference to Fig. 2 relates to a non-rigid image 
registration by an iterative refinement process based on anatomical landmarks 
combining local teooplate matohing and gray value based measurement of image 
similarity. Let image A denote a floating ima^ which has to be reg^tered wifli a 
reference image B. Starting fixMn few but significant landmarks (selected interactively or 

20 automatically), image A is tiled witii sinoplices Pa spent ^a:Jd^\(d = dimensionality of 
the data set) neighboring landmarks. A global deformation field D is constructed by 
locating the landmarks in image B based on local (for instance a£Gne) deformations in 
tiieir nei^borhoods. Every simplex Pa is transformed into simplex Pa =D (P^ and the 
similarily of tiie gray values of imaged lying in and the gray values of imaged lying 

25 in Pb is estimated using, for example, mutual information or cross-correlation. If Pa and 
Pb are not sufBciently sunilar, a new landmark is located in Pa, and Pa is divided into 

smaller shnplices. In this way, tiie tiling is iteratively refined until a desired 
similarity is achieved, resulting in a tiling grid tiiat automatically ad^ itself to local 
non-linearities and anatomical structures. The iteration maybe stopped when die 

30 similarity measure is sufficientiy good in all sinoplices or when a maxinmm number of 
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refinement steps are reached. The resulting global deformation field is then used to 
register the given images. The method is open to adjust the quaHfying fiinction for 
selecting the landmarks, the local similarity measure for matching the landmark- 
centered templates, the global similarity measure for estimating the similarity of the 
5 simplices, and the type of transformation used for modeling tbe non-linear deformations 
to fhe modalities, anatomical domains and contrast mechanisms involved. This will be 
described in ftrther detail with reference to Fig. 2. 

In Fig. 2, after the start in step SI, where, for example, the images to be registered are 
10 acquired, in step S2 a set (a plurality) of significant landmarira {Z, } is selected in both 
hnages A aadB, which can be unambiguously assigned or allotted to each other, Le. 
which unambiguously conespondto each oflier. A significaace of the landmarks maybe 
hereby determined by means of a pre-determined qualifying ftnction. With these 
landmarks Z/, a smq)lex tiling {i>} of the images is determined. For a two-dimensional 
15 image, the simplices are triangles, and for a three-dimensional image, the simpUces are 
tetrahedrons. This is shown m further detail in Fig. 3, depicting image A or image B, 
where five landmarks Li to Ls have been selected- As may be taken fiom Fig. 3, fhe 
landmarks are connected to each other by a line, such that simpHces (here triangles) are 
formed, which tUe the hnage region. This may also be referred to as sinq)lex tiling. 



"20- 



25 



As indicated above, the selection of the landmarks Li may be performed automatically 
or interactively by an operator. For performing such landmark selection automatically, 
the above qualifying fonction may be ^Ked, for exanotple, detecting the brightest spots 
in the images or a certain gradient in the image. 

Then, m the subsequent step S3, local deformation fields A (U Ut) are determined for 
each landmark Z,. hi other words, surrounding each landmark X,, a local deformation 
field A (U Ut) is determined in flie vicinity or region Ut surrounding the respective 
landmark Zj. 



30 
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Form, size and/or dimensions of the region Ui is set such that there is sufficient 
structure, i.e. sufficient image energy in the region Ui allowing to unambiguously 
determine the local deformation. Furthermore, size, form and/or dimension of Ui is 
selected such that the local deformation may be sufficiently approximated by an afiHne 
5 transformation. 

This is shown in further detail in Fig. 4, showing regions C/i to Us surrounding the 
respective landmarks Xi to is . As maybe taken from Fig. 4, the shape of the regions U\ 
to Us is circular. However, it should be noted that also other shapes, such as rectangles 
10 may be applied. Within these regions U\ to Usj the respective local deformation fields 
Di (ii, U^)^oDs (Ls, Us) are determined. 

In the subsequent step S4, the local deformation fields A U^ are expanded to a 
global deformation field on area L In other words, Ihe local deformation fields A O^u 
13 C/^ are extended to a global deformation D onto the whole image area/in image A. 
According to an exemplary embodiment, by varying the way the local deformation 
fields Di (Lu U) are expanded to a global deformation field A the method naay be 
adapted to the respective application. For example. Gauss fimctions, polynomial 
fonctions, elastic fonctions or splines maybe used to perform such expansion. 

20 

In the subsequent step S5, a similarity = 5'(5|^(p^),£)(^|^(^)) is determined. In other 

words, in each tile, region or part of the tiling, a similarity Si for expressing a 

similarity between the reference image ^Di^d transfonned floating image 

D{A) |^(^ J or between the transformed reference image and the floating image. 

25 Advanta^ously, the similarity value is selected such that it increases with an increasing 
conformity between the images and is monotonic to a maximimi value. Then, in Ibe 
subsequent step S6 it is determined whether there is a simplex i^- with i^-^ Z . 

A pie-selected value S of the similarity value S detemsines in which parts of the tiling 
30 {i* } i.e. in which simplex, the conformity between the reference image B and the 
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transformed floating image A is not yet sufficient In case it is determined in step S6 that 
there is a sinqjlex Pj wilh S(P^ < E , then, the method continues to step S7, where a 
refboiement is performed. Otherwise, the method continues to step S8, whete it ends. 

5 In step S7, in the simplex/^ with S'(7y<E , i.e. in the simplex to be refined, anew 
landmark is selected. According to an exemplary embodiment of the present invention, 
shown in Fig.5, the new landmark Z<y is selected in accordance with the qualifymg 
function such that it is lie most significant landmark Z' in the simplex Pj . This 
landmark is added to the set of landmarks {Z J = {Z J . With the new IfttiHmflrV Le, 

10 the simplexi^ is tiled. Here, Le is connected to each vertex (landmarks Z2,Z4, Z5) by a 
straight line. By this, for a two-dimensional image, three new triangles are generated 
and for a three-dimensional image, four new tetrahedrons are generated. In general: (d -K 
1) new simplices are generated for d special dimensions. In the set of simplices {fj }, the 
simplex Pj is replaced by the + 1) new simplices. Then, with the new set of 

15 landmarks {zj and set of simplices {P^} the iteration returns to step S3, wherein, for 
the pre-existing landmarks and the pre-existing regions C7i, the local deformations A are 
maintained. The metiiod iteratively contlimes imtil there is no simplex left with iS^ < Z - 

— Figs.-5-and^6-depict-the-fiu1her-proGessing-aGGordin^ 

20 present invention, as described by steps S3 to S7. In Fig. 5, the simplex Pj with iS^- < L 
is hatched. For this hatched simplex Pj, defined by landmarks Z2, Z4 and Z5, where the 
similarity value is less than a threshold value, a new lanHtnarV Le is selected. Preferably, 
as indicated above, fhe landmark Z6 is a significant or the most significant landmark in 
the region/^*. Then, by connecting Z6 to the landmarks Z^ Z4 and Z5 of i^, three new 
25 simplic^ are generated Then, a local defonnation field Z>6(Z^C/$^ is detenninedm^ 
the local defonnation fields are expanded to a global deformation field. Then, similarity 
values are determined for each of these three new triangles, fii case there is a triangle 
(here in Fig. 5 the triangle Pp), which similarity value is less than the jire-set tinreshold 
value, i.e. Sp<Y»,Si new landmark Z7 is selected in the triangle Pp as shown in Fig. 6. 
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The method iteratively continues until the similarity value for each triangle exceeds or 
equals the threshold value. 

An application example of the above method is depicted in Fig. 7. In Fig. 7, two MR 
5 images of the respiratory system are shown in an exhaled (A) (left image of Fig. 7); and 
in an inhaled state (S) (right image of Fig. 7). A registration by using the landmarks iio, 
Ln andXi2 does not allow a sufficiently accurate registration of the two images since 
the movement of the diaphragm cannot be sufficiently reconstracted. Due to that, 
iteratively, the new significant landmark X13 has been added. 

Advanta^ously, the above method allows for a very accurate and reliable elastic 
registration of single or multi-modality image data sets. It may allow to improve speed 
and robustness of image registration by concentrating on significant landmarks and 
imag^ areas showing higjily non-linear deformations. Furthermore, it may allow the 

15 significant ranking and matching of landmarks, the local and global quality of 

registration and the ^e of non-linear transformations to be independently optimized 
with respect to the modalities, anatomical areas and ei^ected deformations or 
application areas. It may allow to register very different imaging modalities by 
incorporating into these significance and quality measures a priori knowledge about the 

20 anatomy. Due to the focus on few but significant landmarks in conjunction with its 
automatic and spatial adaptivity to non-linear deformations, the above described 
invention may be very fest and well suited for real time registration, for example, during 
interventional procedures. In medical plications, global elastic registration may be 
required for a variety of clinical applications, where images acquired at diEferent times, 

25 with different modalities or different patients have to be aligned, bnportant examples o:^ 
for example, tumor diagnostics and surgery, where images of different modalities show 
different aspects of the tumor, a comparison of pre- and post-intervention images, 
analysis of time series of medical images, matching of individual images wifli 
anatomical atiases derived from cohort studies etc. However, it should be noted that the 

30 above described invention may also be applied to any two or three-dimensional data 
sets. 
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CLAIMS 



1 . Method of registering a first image and a second image, the method comprising the 
steps of: 

selecting at least one first landmark: in the first image; 

selecting at least one second landmark in the second image; and 
5 registering the first and second images by using a similarity value which relates 

to a similarity of a first region in the fibcst image determined by the at least one first 
landmark and a second region in the second image determined by the at least one 
second landmark; 

wherein the at least one first landmaric corresponds to Ihe at least one second 
10 landmark. 

2. The method of claim 1, 

wherein, for the first image having a first nund^er of dimensions, a second 
number of first landmarks is selected; 
15 wherein, for the second image having the first number of dimensions, the second 

number of second landmarks is selected; 

wheiein the second number of first landmarks determine first vertices of first 
simplices for determining a third number of third regions in the first ixnag^; 

wherein tihie second number of second landmarks determine second vertices of 
20 second simplices for determining the third number of fourth regions in the second 
imag^; 

wherein the seccmd number is tiie fiiret number plus one; 
wherein the first and second landmaiks are selected in accordance with a 
qualifying function; 



25 
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3. The method of claim 2, 

wherein, for each of fhe second number of second landmarks, a local 
deformation field is determined for determining a first global deformation field which 
approximately describes a deformation required to tibe first image for registration onto 
the second image. 

4. The method of claim 3, 

wherein, for each of the third nmnber of third regions, a first similarity value is 
determined relating to a similarity between a respective one of the third regions to a 
respective corresponding one of the fourth regions by using the first global deformation 
field. 

5. The method of claim 4, 

wherein it is determined whether there is a fifth region of fhe third regions which 
first similarity value is less than a preset threshold value; 

wherein, when there is a fifth region, a tiiird landmark is selected in the fifth 
region for determining third simplices in fhe fifth region which determine a plurality of 
sixth regions; 

wherein, when there is a fifth region, a fourth landnoiark is selected in a seventh 
region of fourth regions for determining fourth simplices in the seventh region which 
'^jstemsiaea^l^^ 

wherein fhe third landmark corresponds to fhe fourth landmark such that fhe 
sixth regions correspond to the eighth regions; 

wherein, for each of the sixth regions, a second similarity value is determined 
relating to a similarity between a respective one of fhe sixth regions to a respective one 
of tiie eighth regions by using a second global deformation field which has been refined 
by using a further local deformation field of fhe fhhd landmark. 

6. The method of claim 5, 

wherein fhe method is iteratively repeated until all similarities exceed the preset 
threshold value. 
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7. The method of claim 1. 

wheiein the method is applied in medical imaging to one of CT data sets, MRI 
data sets, PET data sets, SPECT data sets, and ultrasonic imaging data sets. 

5 

8. Image processing device, comprising: 

a memory for storing a first image and a second image; and 

an image processor for registaing the first imag^ and the second image, wherein 

the image processor is adapted to perform the following operation: 
10 selecting at least one first landmark in the first image; 

selecting at least one second landmark in the second image; and 

registering the first and second images by using a similarity value which relates - 

to a similarity of a fibrst region in the first image determined by the at least one first 

landinarlr and a second region in the second image determined by the at least one 
15 second landmark; 

wherein the at least one first landmark corresponds to the at least one second 

landmark. 

9. Coinputer program for registering a first image and a second image, wherein the 
20 computer program causes a processor to perform the following operation when the 

computer program is executed on the processor: 

selecting at least one Srst landmark in the first image; 
selecting at least one second landmark in the second image; and 
registering the first and second images by using a similarity value which relates 
25 to a similarity of a first region in the first image determined by the at least one first 

landmark and a second region in the second image detennined by the at least one 

second landmark; 

wherein the at least one first landmark corresponds to flie at least one second landmark. 
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ABSTRACT 

Elastic image registration 

Image registration very often used to be a tedious task which had to be performed 
manually. According to an exeniplaiy embodiment of the present invention, a non-rigid 
5 image r^tcation by an iterative refinement process based on a combination of 
iatiHiT>aTlc« with similarity values is proposed. Advanta^usly, a veiy fast and robust 
method may be provided. 
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